test(perf): add e2e perf tests for balanced consolidation - #3247

Fixes #N/A

Description

Adds end-to-end performance specs for the Balanced consolidation policy landed in #2962. The specs provide regression coverage for score-driven consolidation decisions on top of the existing basic and wideDeployments suites.

Three artifacts are added under test/:

1. LatencyHarness + LatencySidecar (in test/pkg/environment/common/) is the shared substrate. LatencyHarness scrapes Karpenter's /metrics at phase start and stop and reduces per-series histogram bucket deltas into percentile stats (P50 / P90 / P95 / P99 plus a bucket-truncation rate) and counter deltas over the observation window. LatencySidecar is the on-disk JSON schema written alongside PerformanceReport when OUTPUT_DIR is set. Both are consumed by the new specs so a PerformanceReport and its latency companion pair on disk for offline diff analysis.

2. test/suites/performance/balanced_baseline_marginal_test.go covers two specs. Balanced Baseline It reuses the 1000-pod / 700-pod fixture from basic_test.go, wrapped with the harness so hero histograms (scheduling_decision, voluntary_disruption_decision_evaluation, pods_bound) are captured for both scale-out and consolidation phases. This is the reference distribution the marginal runs compare against; suite-wide WhenEmptyOrUnderutilized applies. Balanced Marginal Move runs two variants that share the same fixture. The small deployment's pod-deletion-cost annotation is the only knob and shifts the pool's total_disruption_cost denominator to push the consolidation score across the 1/k=0.5 threshold. Approved variant runs at cost=0; rejected variant runs at cost=2e9. Both run under ConsolidationPolicyBalanced.

3. test/suites/performance/balanced_churn_heterogeneous_test.go covers churn-chain and heterogeneous NodePool scenarios. The churn-chain spec drives a 400-pod scale-out followed by three scale-in / scale-out rounds under each ConsolidationPolicy; the harness spans the churn window. The heterogeneous spec runs two family-restricted NodePools (KWOK c and m families) with distinct pod-density profiles and observes cross-pool consolidation on scale-down. Per-pool decisions are captured via the same paired-artifact scheme.

The paired-run pattern (WhenEmptyOrUnderutilized first, then Balanced) keeps the second run starting from an AfterEach-clean state so diff analysis stays stable. Neither variant asserts an exact karpenter_consolidation_moves_total count; KWOK timing blurs which candidates land in a given round, so the soft directional check hasScoreSeriesForDecision only logs when the expected decision series is absent.

Provider-specific threshold overrides continue to work through the KARPENTER_PERF_THRESHOLDS env var from #3165. The threshold-lift companion for the existing basic and wideDeployments suites is in #3245.

How was this change tested?

make verify passes locally. make presubmit passes including vulncheck. Compile is clean under go test -c ./test/suites/performance/... and unit tests in ./test/pkg/environment/common/... pass. The e2e specs themselves cannot be exercised outside the kind-perf-e2e workflow; fork CI will verify that path after push. Threshold values (for the new specs) were chosen with headroom over observed samples rather than from a full re-baseline capture; the rationale for the accompanying threshold-lift on the pre-existing suites is in #3245.

AI Disclosure

I used an LLM to parallel program with me on this. That included drafting the PR description, running the rebase against upstream/main, and the three small cleanup refactors (histogram reducer complexity, LatencySidecar hoist, and magic-number naming). All benchmark spec logic was authored by me previously; the LLM did not write the perf spec bodies.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.